kv cache

Practical guide to speed up Ollama on Linux: store models on NVMe and keep them hot (OLLAMA_KEEP_ALIVE, OLLAMA_MAX_LOADED_MODELS), pick sensible quantization (q4_K_M/q5_1/q8_0/f16), offload layers to GPU and tune num_gpu_layers, num_batch, num_ctx, set sane concurrency and OLLAMA_MAX_QUEUE, warm favorite models, and benchmark tokens/sec via the HTTP API. Yields snappier first tokens and higher throughput.

Practical Linux guide to speed up Ollama: explains why tuning matters and gives copy-paste steps to boost tokens/sec and cut latency - pick the right model/quant, enable CUDA/ROCm GPU offload and adjust num_gpu, set threads and context (num_thread/num_ctx), keep models warm, store GGUFs on fast NVMe via OLLAMA_MODELS, benchmark with curl+jq and monitor with htop/nvtop - plus quick CPU/GPU recipes and common pitfalls.